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STUDY  TITLE: 

* 

AN  ASSESSMENT  OF  THE  PRACTICALITY  OF  A TOTAL  SHIP  OPEVAL 


STUDY  PROJECT  GOALS: 

To  identify  and  analyze  the  considerations  that  pertain  to  the  concept  of 
conducting  total  ship  Operational  Evaluations  (OPEVALs)  as  part  of  the  ship 
acquisition  process  and  to  document  in  the  study  report  these  considerations 
and  the'  conclusions  that  can  be  drawn  for  future  ship  programs. 


STUDYj  REPORT  ABSTRACT: 

The  overall  purpose  of  the  study  project  v/as  to  assess  the  practicality 
of  the  total  ship  OPEVAL  for  a conventional  (production  vice  R&D)  ship  pro- 
gram. This  report  contains  an  overview  of  the  Navy  T&E  actions  that  were 
taken  to  implement  the  policies  of  DOD  Directive  5000.3.  A detailed  analysis 
of  the  ship  acquisition  T&E  process  is  provided,  from  which  a model  is  gener- 
ated to  serve  as  a framework  for  analyzing  the  marginal  utility  of  adding  a 
total  ship  OPEVAL  to  this  process.  Three  considerations  emerge  which  lead  to 
the  conclusion  that  an  OPEVAL  would  be  impractical  in  all  cases  (except  the 
R&D  case,  which  was  not  considered) : (1)  there  are  inherent  constraints  and 

artificialities  in  such  an  exercise;  (2)  the  objectives  are  already  achieved 
under  the  existing  ship  T&E  process,  and  (3)  there  would  be  a negligible  • 
return  on  investment. 

A recommendation  is  made  that  DOD  Directive  5000.3  be  revised  to  reflect 
the  conclusions  of  this  report  and  that  two  Navy  T&E  guidance  manuals  be  re- 
vised to  reflect  the  conceptualization  of  the  ship  acquisition  T&E  process 
described  in  the  report.  Recommendations  are  made  for  several  follow-on 
study  projects. 
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This  study  project  report  represents  the  views,  conclusions  and 
recommendations  of  the  author  and  does  not  necessarily  reflect 
the  official  opinion  of  the  Defense  Systems  Management  School  or 
the  Department  of  Defense. 


EXECUTIVE  SUMMARY 


This  report  describes  the  results  of  a study  project  in 
which  the  author  investigated  the  practicality  of  conducting 
a total  ship  Operational  Evaluation  (OPEVAL)  as  part  of  a 
Navy  ship  acquisition  program.  The  OPEVAL  has  become  a very 
useful  tool  which  the  Navy  has  used  for  the  conduct  of 
Initial  Operational  Test  and  Evaluation  (IOT&E)  of  its  new 
systems.  Since  the  time  that  the  Office  of  the  Secretary  of 
Defense  (OSD)  first  issued  its  T&E  policies  in  1971,  there 
has  been  a disagreement  between  OSD  and  the  Navy  on  whether 
or  not  the  OPEVAL  should  be  conducted  on  ships  in  conventional 
(i.e.  production  vice  R&D)  ship  programs.  The  Navy  has  taken 
the  position  that  it  should  not. 

In  this  study  project,  the  reasons  why  the  Navy  has  taken 
this  position  were  investigated.  Lessons  learned  from  ship 
IOT&E  programs  conducted  within  the  past  five  years  were 
assessed.  Interviews  were  conducted  with  T&3  managers  from 
current  ship  acquisition  projects  and  with  personnel  from 
the  Office  of  OSD's  Deputy  Director  for  Test  and  Evaluation. 

The  study  project  concluded  that  the  conduct  of  a total 
ship  OPEVAL  would  be  impractical  in  all  cases  (except  the 
R&D  case,  which  was  not  considered).  Three  considerations 
are  discussed  in  detail  that  support  this  conclusion: 

1.  The  inherent  constraints  of  the  test  exercise 
would  in  themselves  introduce  uncertainties  to 
and  undermine  the  confidence  in  the  validity  of 
the  results; 
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2.  the  objectives  of  an  OPEVAL  as  applied  to  a ship 
program  are  already  achieved  under  the  existing 
Navy  ship  T&E  process;  and 

3.  even  assuming  that  some  additional  knowledge  would 
be  gained  from  the  OPEVAL,  it  is  doubtful  that 
there  would  be  a significant  return  on  investment. 
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SECTION  I 


INTRODUCTION 

Purpose  of  the  Study  Project 

The  purpose  of  this  study  project  was  to  investigate  the 

practicality  of  conducting  a total  ship  Operational  Evaluation 

(OPEVAL)  as  part  of  a Navy  ship  acquisition  program.  For 

over  a decade  the  OPEVAL  has  been  used  by  the  Navy  as  a tool 

for  assessing  the  suitability  of  new  systems  prior  to  their 

introduction  to  the  Fleet.  When  Deputy  Secretary  of  Defense 

David  Packard  first  introduced  the  new  DOD  Test  and  Evaluation 

(T&E)  policies  to  the  Services  in  1971  (1:1-2)  \ the  OPEVAL 

was  ideally  suited  as  a mechanism  for  the  conduct  of  Initial 

Operational  Test  and  Evaluation  (IOT&E) . During  the  next  two 

years,  as  the  new  DOD  directive  that  would  aefinitize  and 

document  the  T&E  policies  was  prepared,  personnel  from  the 

Office  of  the  Secretary  of  Defense  (OSD)  and  the  Navy  had 

many  discussions  on  how  these  policies  would  be  applied  to 

ships.  A specific  area  of  disagreement  was  the  utility  of 

conducting  an  OPEVAL  on  a total  ship.  The  issue  was  not 

totally  resolved.  The  DOD  directive,  when  finally  issued, 

reflected  this  by  inclusion  of  the  following  statement: 

"For  all  new  classes,  continuing  phases  of  OT&E  on 
the  lead  ship  will  be  conducted  at  sea  as  early  in 


This  notation  will  be  used  throughout  the  report  for 
sources  of  quotations  and  major  references.  The  first  number 
is  the  source  listed  in  the  References  section;  the  second 
number,  if  provided,  is  the  page  within  that  source. 
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the  acquisition  process  as  possible  for  specified 
systems  or  equipments  and,  if  required,  full  ship 
operational  evaluation  to  the  degree  feasible. " 

(underline  added)  ( 2 : 5 } 

The  specific  goals  of  this  study  project  were  to  study 
the  reasons  why  the  Navy  objected  to  the  concept  of  a total 
ship  OPEVAL  in  that  1971-72  time  frame,  to  investigate  any 
new  knowledge  the  Navy  has  gained  since  that  time  from  the 
actual  conduct  of  ship  IOT&E  programs,  and  to  make  recommenda- 
tions for  future  ship  programs  as  well  as  to  suggest 
possible  changes  to  the  DOD  T&E  directive. 

Limitations  of  the  Project 

As  will  be  explained  in  Section  II,  separate  policies 
apply  to  programs  where  the  lead  ship  is  constructed  and 
tested  as  ar.  effort,  in  contrast  to  the  more  conventional 
approach  of  building  the  lead  ship  as  a' production  effort. 

In  the  case  of  the  R&D  ship,  there  is  a requirement  for  the 
conduct  of  an  OPEVAL.  While  some  of  the  considerations  that 
apply  to  a .conventional  ship  OPEVAL  might  be  applied  here 
also,  this  topic  was  not  included  in  this  study  project. 

Study  Project  Methodology 

While  many  of  the  considerations  that  pertained  to  this 
topic  during  the  1971-1972  time  frame  are  well  documented, 
much  has  been  learned  but  little  written  since  that  time.  In 
investigating  what  has  been  learned,  it  was  therefore 
necessary  not  only  to  survey  the  literature  on  the  subject, 
but  also  to  conduct  interviews  with  key  participants  in  ship 
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IOT&E  programs  that  have  recently  been  planned  and/or  conducted 
and  with  personnel  from  the  Office  of  OSD's  Deputy  Director 
for  Test  and  Evaluation.  The  list  of  pertinent  references 
as  well  as  personnel  interviewed  is  contained  in  the 
References  section  of  this  report. 

Organization  of  the  Report 
The  information  gathered  for  this  report  has  been 
organized  around  the  flow  of  a typical  ship  acquisition 
T&E  program,  with  an  emphasis  towards  highlighting  the 
manner  in  which  this  process  complies  with  the  spirit  and 
intent  of  the  DOD  directive  on  T&E.  Section  II  provides 
background  information  on  the  development  of  Navy  T&E 
policies  to  implement  the  DOD  directive  and  on  the  peculiar- 
ities of  ship  acquisition  T&E.  Section  III  is  a detailed 
analysis  of  the  ship  acquisition  T&E  process.  It  describes 
the  key  activities  that  occur  during  the  phases  of  ship 
acquisition  T&E  and  presents  a model  which  serves  as  a 
framework  for  considering  the  marginal  utility  of  adding  a 
total  ship  OPEVAL  to  the  T&E  process.  Section  IV  presents 
the  considerations  that  emerged  from  the  study  project  which 
lead  to  the  conclusion  of  the  impracticality  of  the  total 
ship  OPEVAL.  Section  V summarizes  the  conclusions  drawn  and 
includes  some  recommendations  for  the  Navy  and  OSD  to  follow 
based  on  those  conclusions.  ' 
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SECTION  II 


OVERVIEW  OF  THE  NAVY  T&E  POLICIES  AND  PROCEDURES 

Background 


Test  and  Evaluation  (a&E)  now  plays  a key  role  in  the 
weapons  acquisition  process.  When  the  theoretical  benefits  did 
not  accrue  from  the  total  package  procurement  concept  instituted 
by  Secretary  of  Defense  Robert  McNamara  in  the  early  1960 's, 
new  approaches  were  studied  and  the  milestone  procurement  policy 
was  adopted  by  Deputy  Secretary  of  Defense  David  Packard  in  the 
early  1970*s.  {3:1-7}  Under  this  policy,  instead  of  a one-time 
commitment  to  an  entire  research,  development  and  production 
program,  funds  and  resources  are  committed  incrementally.  The 
acquisition  process  is  thus  separated  into  phases  by  discrete 
decision  points  at  which  the  progress  of  the  program  is  reviewed 
and  a decision  is  made  on  whether  or  not  to  proceed.  One  of  the 
critical  inputs  to  these  decisions  is  the  result  of  T&E,  particu- 
larly Operational  T&E.  Operational  T&E  is  defined  as  T&E  con- 
ducted by  the  cognizant  Service's  designated  independent  agent 
in  an  operational  environment  for  the  purposes  of  assessing  mili- 
tary suitability  and  effectiveness.  The  Navy's  OT&E  agent  is 
the  Commander,  Operational  Test  and  Evaluation  Force  (CPTEVFOR) , 
who  repcrts  directly  to  the  Chief  of  Naval  Operations  and  is 
independent  of  the  developing  activities  {Navy  Material  Command), 
and  the  user  activities  (the  Fleet). 


A 


The  OPEVAL 


The  most  common  and  formal  vehicle  that  COMOPTEVFOR  uses  to 
perform  his  function  is  an  Operational  Evaluation  (OPEVAL).  (4:6) 
An  OPEVAL  is  a project  assigned  by  the  Chief  of  Naval  Operations 
to  OPTEVFOR  to  conduct  T&E  of  a specific  system  in  its  projected 
operational  environment  for  the  sake  of  providing  an  independent 
assessment  of  that  system's  military  effectiveness  and  suita- 
bility. Through  the  OPEVAL,  COMOPTEVFOR  assesses  whether  or  not: 

(1)  The  system  or  equipmert  functions  in  an  operationally 
satisfactory  manner  and  performs  reliably  and  effectively  in 
accordance  with  progrf  n objectives  in  realistic  operational  con- 
ditions. 

(2)  The  system  can  be  effectively  operated  and  maintained 
by  the  level  of  personnel  skill  anticipated  to  be  available 
under  service  conditions. 

(3)  There  is  reasonable  indication  that  logistic  support- 
ability  in  a deployed  status  is  feasible. 

(4)  Ali  test  questions  germane  to  a production  decision 
are  adequately  examined. 

Since  most  Navy  systems  and  equipments  are  destined  to  be  instal- 
led on  ships  or  aircraft,  the  conduct  of  an  OPEVAL  of  a new  sys- 
tem frequently  involves  an  actual  temporary  installation  of  that 
system  in  an  active  fleet  ship  or  aircraft. 

In  a typical  Navy  acquisition  program,  the  OPEVAL  is  the  x 
"final  exam"  of  the  R&D  phase.  If  it  is  successful,  it  will  re- 
sult in  a recommendation  from  OPTEVFOR  to  the  Chief  of  Naval 
Operations  that  the  acquisition  program  proceed  from  R&D  to  pro- 
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duction  and  fleet  introduction.  (This  T&E  input  is  only  one 
input  - although  usually  the  most  critical  - to  the  production 
approval  decision.  The  final  decision  will  also  depend  on  other 
factors  such  as  cost,  schedule,  and  military  need.)  The  conduct 
of  an  OPEVAL  at  the  end  of  the  R&D  phase  is  a critical  Navy 
mechanism  for  application  of  the  DOD  policy  c "try-before-buy." 
Figure  1 is  the  schedule  of  a Navy  gun  system  T&E  program  which 
quite  typical  and  demonstrates  the  process  just  described. 
This  system  is  the  UK  75  Gun  and  MK  92  Fire  Control  System  which 
will  initially  be  installed  on  guided  missile  frigates  of  the 
FFG-7  Class  and  on  patrol  combatant  hydrofoil  ships  of  the  PHM-1 
Class.  A prototype  version  of  this  gun  system  was  temporarily 
installed  onboard  USS  TALBOT  (FFG-4)-  from  September  1974  to 
November  1975.  The  OPEVAL  was  successfully  conducted  from  April 
to  June  1975  and  CNO  subsequently  approved  the  system  for  pro- 
duction on  8 September  1975.(5:1) 

T&E  of  Ships 

From  the  earliest  studies  of  the  "try-before-buy"  concept, 
applying  this  policy  to  Naval  ships  has  remained  somewhat  of  an 
anomaly.  The  pure  application  would  be  to  use  the  lead  ship  of 
a class  as  a prototype  for  the  purpose  of  conducting  an  OPEVAL, 
prior  to  approving  construction  of  the  "follow  ships."  However, 
for  most  programs  this  is  not  practical  because  of  the  time 
associated  with  the  design  and  construction  of  a ship.  Figure  2 
is  the  schedule  for  construction  of  the  lead  ship  of  the  FFG-7 
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Class  guided  missile  frigates.  The  FFG-7  program  is  typical 

of  a Navy  combatant  ship  program  and  is  the  first  ship  program 

to  be  structured  to  conform  to  the  new  DOD  milestone  procure- 
2 

ment  process.  Note  that  the  lead  ship  is  scheduled  for  de- 
livery in  June  1977.  Because  of  the  time  required  for  crew 
training,  qualifications  and  overall  shakedown,  the  ship  would 
not  be  ready  to  conduct  an  exercise  such  as  an  OPEVAL  for  almost 
a year.  Adding  to  that  the  time  for  the  OPEVAL  itself,  the 
analyses  of  the  results,  and  preparation  of  the  report,  it  would 
be  almost  1977  before  any  OT&E  data  from  the  lead  ship  would  be 
available  for  a production  decision.  Estimating  42  months  from 
time  of  contract  award  to  delivery  of  the  first  follow  ship, 
there  would  then  be  a five  year  delay  between  delivery  of  the 
first  ship  and  delivery  of  the  first  follow  ship,  if  the  program 
were  structured  to  use  the  lead  ship  as  a prototype. 

It  was  agreed  between  OSD  and  the  Navy  that  for  most  ship 
programs,  the  spirit  and  intent  of  the  "try  before  buy"  policy 
could  be  achieved  without  delaying  approval  for  follow-ship 
production  until  OT&E  could  be  conducted  on  the  lead  ship.  The 
normal  combatant  ship  acquisition  usually  includes  the  use  of 
other  installations  on  which  sufficient  operational  T&E  data 
can  be  generated  to  provide  assurances  of  the  suitability  and 
effectiveness  of  the  overall  ship.  These  installations  are  (1) 
the  temporary  shipboard  installations  (surrogate  platforms)  used 
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for  OPEVALs  of  individual  unproven  shipboard  systems  and  (2) 
the  land  based  test  sites  frequently  constructed  for  the  inte- 
gration of  shipboard  subsystems  prior  to  installation  in  a lead 
ship.  (Most  major  combatant  ships  have  been  of  a complexity  to 
warrant  construction  of  a land  based  test  site  for  the  combat 
system,  and  several  of  them  for  the  propulsion  system.)  This 
procedure  for  ship  acquisition  OT&E  is  documented  in  DOD  Direc- 
tive 5000.3  on  T&E. 

There  is  one  not&ole  exception  to  this  rule.  When  a ship 
design  involves  a major  technological  advance  in  the  hull  or 
propulsion  design,  the  lead  ship  is  designed,  constructed,  and 
tested  in  its  entirety  as  an  R&D  effort.  Such  ships  must  under- 
go extensive  operational  testing  prior  to  the  commitment  to  the 
follow-on  production  of  ships.  Testing  covers  all  the  extreme 
conditions  to  which  the  ships  will  be  subjected  in  the  Fleet  in 
order  to  assure  that  this  new  platform  can  perform  its  designated 
mission.  The  two  key  Navy  programs  currently  following  this 
approach  are  the  patrol  combatant  hydrofoil  ship  (PHM)  and  the 
Surface  Effects  Ship  (SES) . The  lead  hydrofoil  ship,  the  PHM-1, 
will  undergo  a total  ship  OPEVAL  in  late  Spring  of  1976.  Unlike 
the  FFG-7  program,  the  follow-ship  production  contract  will  not 
be  awarded  until  after  this  OPEVAL  and  subsequent  approval  at 
DSARC  III.  The  structure  of  the  PHM-1  program  is  depicted  in 
Figure  3,  where  it  can  be  contrasted  to  the  more  conventional  , 
schedule  of  the  FFG-7  program  in  Figure  2. 

The  Surface  Effects  Ship  program  includes  planning  for  a 
full  ship  prototype  in  the  2200  to  3000  ton  range,  the  key 
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purpose  of  which  will  be  developmental  and  operational  T&E.  Un- 
like PHM-1,  which  will  eventually  be  deployed  as  a Fleet  unit, 
the  SES  prototype  wilJ  ho  a test  vehicle  only,  constructed  with 
no  intention  of  actual  fleet  usage. 
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SECTION  III 


THE  SHIP  ACQUISITION  T&E  PROCESS 
Design  and  T&E:  Two  Parallel  Processes 

The  T&E  in  an  acquisition  program  is  tailored  to  reduce 
the  risks  inherent  in  the  design.  For  a given  program,  there- 
fore, the  T&E  process  closely  parallels  the  design  process. 
This  is  a fundamental  concept  to  understand  when  considering 
the  anomalies  associated  with  T&E  for  ship  acquisition  as 
compared  to  that  for  acquisition  of  other  weapons  systems. 

The  anomalies  are  the  same  ones  that  are  inherent  in  the  over- 
all ship  design  process  itself. 

The  key  anomaly  referred  to  is  that  from  an  operational 
performance  perspective  the  ship  is  basically  only  a platform 
for  other  systems.  For  this  reason,  most  of  the  management 
concepts,  theories  and  approaches  that  can  be  applied  to 
other  weapons  acquisition  programs  cannot  be  given  a wholesale 
application  to  ship  acquisition  programs.  It  is  only  within 
the  last  decade  that  ships  have  begun  to  take  on  some  of  the 
characteristics  of  unique  systems  in  themselves.  This  is 
due  largely  to  the  fact  that  the  design  and  operation  of  the 
shipboard  systems  have  become  increasingly  interdependent 
with  each  new  ship  class  in  an  effort  to  reduce  the  response 
time  required  to  perform  its  missions. 

Program  Initiation 

The  acquisition  program  for  a new  weapon  system  is  ' 

initiated  in  response  to  a clearly  defined  operational  need 
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(e.g.,  a specific  threat)  which  cannot  be  satisfied  by  exist- 
ing military  or  commercial  hardware.  (3:3)  (6:5)  An  Opera- 
tional  Requirement (s)  is  first  written  to  provide  a concise 
statement  of  operational  needs.  A Development  Proposal  is 
then  prepared  which  presents  alternatives  and  tradeoffs  for 
achieving  the  particular  capabilities  needed  to  meet  the  identi- 
fied operational  requirement.  If  this  proposal  is  approved, 
development  begins.  Assuming  that  the  development  efforts  are 
successful  and  the  program  receives  approval  to  proceed  at  its 
program  initiation  and  full-scale  development  milestones  two 
of  the  end  products  will  be  (1)  a prototype  for  the  conduct  of 
DT&E  and  OT&E  and  (2)  a detail  specification  for  use  in  the 
production  contract.  The  T&E  requirements  are  developed  in 
parallel  with  this  design  process:  operational  T&E  requirements 

from  the  Development  Proposal,  and,  later,  production  acceptance 
T&E  requirements  from  the  contract  specifications. 

In  contrast  to  this  "top  down"  direction  of  the  develop- 
ment of  design  and  T&E  requirements  for  other  weapon  systems, 
those  requirements  for  a ship  are  generated  more  from  the 
"bottom-up,"  beginning  with  the  early  stages  of  assembly  of 
individual  equipments  and  systems  and  ending  with  at-sea 
exercises  of  •’.he  ship.  The  ship's  suite  of  systems  is  a 
composite  of  the  latest  types  of  systems  needed  to  perform 
the  mission  of  that  particular  ship  type.  Some  of  these  of 

\ 

necessity  will  still  be  developmental  at  the  time  of  their 
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selection.  However,  an  attempt  is  made  to  insure  that  all 
of  them  can  be  subjected  to  an  OPEVAL  and  be  approved  for 
service  use  by  the  time  they  are  actually  installed  during 
the  ship  construction  period.  Thus,  for  the  most  part,  the 
design  and  development  of  shipboard  systems  precede  that  of 
the  ship  and  are  somewhat  independent  of  it.  The  ship  itself 
is  not  designed  as  a unique  system. 

The  timing  of  when  to  initiate  a new  ship  acquisition 
program  is  usually  not  related  to  a specific  operational 
threat,  but  rather  to  a need  for  platforms  to  replace  ships 
that  have  reached  obsolescence,  thereby  maintaining  or 
increasing  force  levels  (the  notable  exception  to  this  being 
R&D  ship  programs  which  are  initiated  in  order  to  force  or 
capitalize  upon  technological  advancement  in  the  state-of- 
the-art,  such  as  the  current  hydrofoil  or  surface  effects 
ship  R&D  efforts) . 

T&E  of  the  Individual  Shipboard  Systems 

A ship,  as  a platform  for  different  types  of  systems,  is 
designed  to  accomplish  many  missions.  A destroyer,  for 
example,  has  many  and  varied  missions  including  anti-air 
warfare,  anti-submarine  warfare,  shore  bombardment,  search 
and  rescue,  blockade  and  surveillance,  and  harassment.  Each 
shipboard  system  plays  a role  in  contributing  to  the 
accomplishment  of  one  or  more  of  these  missions.  As  mentioned 
before,  the  design  and  development  of  individual  systems 
precede  and  are  virtually  independent  of  specific  ships.  The 


14 


test  of  individual  systems  to  meet  certain  threats  is  there- 
fore also  generally  independent  of  specific  ships.  A given 
gun  system,  for  example,  will  be  used  on  many  types  of  ships 
from  destroyers  to  replenishment  oilers.  To  perform  an  OPPJVAL 
of  that  gun,  it  will  be  temporarily  installed  on  one  active 
fleet  ship  - perhaps  a destroyer  - for  testing  in  an 
operational  environment.  However,  the  results  of  the  OPEVAL 
on  that  one  type  ship  will  be  sufficient  to  make  a decision 
for  approval  to  produce  that  system  for  all  of  its  prospective 
platforms.  Almost  all  of  the  operational  risk  and  uncertainty 
in  the  acquisition  of  that  gun  are  inherent  in  the  system  it- 
self. The  suitability  and  effectiveness  of  the  gun  with 
regard  to  the  role  it  will  play  on  the  replenishment  oiler 
can  be  almost  totally  demonstrated  on  the  destroyer  platform. 
In  other  words,  the  marginal  operational  risk  and  uncertainty 
related  to  its  installation  in  a specific  ship  platform  and 
its  integration  with  the  other  systems  on  that  platform  is 
negligible. 

The  ship  design  process,  then,  as  it  relates  to  the 
particular  shipboard  systems,  consists  of  integrating  the 
selected  systems  in  some  optimum  fashion  to  insure  mutual 
compatibility  and  to  reduce  response  time  in  accomplishing 
the  ship's  missions,  as  well  as  accommodating  the  physical 
constraints  of  the  ship  as  a platform. 

The  Navy  T&E  process  for  a conventional  ship  acquisition' 
program  is  based  on  the  fact  that  the  operational  risk 
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associated  with  the  ship  platfc  -self  is  minor  when  com- 
pared to  that  attributable  tc*  the  individual  systems.  The 
Navy  has  made  a major  policy  cr>mmJ ' "Qnt  to  demonstrate  sys- 
tem operational  effectiveness  and  suitability  independent  of 
and  prior  to  permanent  installation  on  specific  ship  platforms. 
When  a lead  ship  is  constructed,  assuming  that  all  of  the  sys- 
tems have  successfully  completed  their  development  and  opera- 
tional T&E,  including  their  own  OPEVALs,  there  is  little 
operational  risk  that  remains  and  thus  minimal  operational  T&E 
to  be  conducted  on  the  lead  ship  itself.  That  which  does  remain 
to  be  conducted  is  not  consequential  enough  to  warrant  delay 
of  a decision  to  build  "follow  ships"  until  it  is  completed. 

The  Navy's  commitment  is  a very  conscious  one,  with  far 
reaching  implications  for  the  entire  spectrum  of  T&E  policies 
and  procedures.  This  can  be  appreciated  if  one  considers  the 
hypothetical  possibility  that  the  Navy  approach,  in  contrast, 
could  be  to  delay  the  development  and  operational  T&E  of  the 
systems  until  they  are  installed  on  the  lead  ship,  and  thereby 
actually  use  the  lead  ship  itself  as  a total  prototype. 

Land  Based  Test  Sites 

Land  based  test  sites  are  another  important  aspect  of 
ship  design  and  T&E  processes  which  further  demonstrate  the 
Navy's  commitment  to  early  reduction  of  development  and 

\ 

operational  risks  prior  to  ship  construction.  Many  of  the 
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shipboard  systems  - particularly  in  the  combat  systems  area  - 
have  become  very  operationally  interdependent  in  ships 
designed  during  the  past  ten  years.  This  interdependence  has 
introduced  an  additional  degree  of  development  and  operational 
risk  in  integrating  these  systems.  The  Navy  has  made  a 
conscious  decision,  however,  not  to  defer  the  reduction  of 
this  integration  risk  to  shipboard  installation.  Land  based 
test  sites  (LBTSs)  have  been  constructed  for  major  combatant 
ship  programs  specifically  for  the  design  and  T&E  of  system 
integration  prior  to  shipboard  installation.  Thus,  the  Navy 
commitment  to  minimize  the  development  and  operational  risk 
associated  with  the  ship  platform  has  been  preserved.  A 
detailed  description  of  the  FFG-7  program’s  Combat  System 
Land  Based  Test  Site  located  in  Long  Island,  New  York,  as  well 
as  the  DT&E  and  IOT&E  program  conducted  there,  is  contained 
in  Appendix  A.  Therein  the  extent  to  which  the  Navy  goes  to 
minimize  operational  risks  and  uncertainty  prior  to  ship  con- 
struction can  be  seen. 

The  Ship  Construction  Period 
After  integration  and  testing  of  the  shipboard  systems 
at  the  land  based  test  site(s),  the  major  design  and  T&E 
activity  moves  to  the  shipyard.  Here,  for  a time  period  of 
three  to  five  years,  detailed  drawings,  construction  plans, 
work  orders,  and  inspection  plans  are  developed  and  executed 
as  the  ship  platform  is  constructed  and  her  systems  are  installed. 
During  the  last  year  the  critical  efforts  center  about  the  test 
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program  as  each  p'ece  of  equipment  is  lit  off,  tested,  mated 
with  its  interfaces,  and  retested  at  the  next  level  of  inte- 
grated operability.  A very  extensive  Integrated  Test  Package 
is  developed  and  its  tests  are  conducted  to  insure  that  all 
of  the  systems  are  properly  installed  and  functioning  within 
prescribed  tolerance  levels  prior  to  the  at-sea  demonstration 
for  the  Navy  Acceptance  Trials.  (7:2)  (8:27)  T&E  during  this 

period  is  neither  development  nor  operational,  but  is  categor- 
ized as  production  acceptance  T&E  (PAT&E) . (4:9) 

The  Post  Delivery  Tests  and  Trials  Period 
Finally,  after  Acceptance  Trials  and  delivery  to  the 
Navy,  the  ship  undergoes  a one  year  period  of  post  delivery 
tests,  trials,  and  training  exercises  prior  to  deployment. 

Most  of  this  effort  is  directed  towards  individual  system  certi 
fications  and  a "shakedown"  of  the  ship  and  her  crew.  Towards 
the  end  of  that  time  frame,  a second  set  of  Navy  trials  (Final 
Contract  Trials)  is  conducted  by  the  Board  of  Inspection  and 
Survey  to  verify  the  correction  of  material  deficiencies  pre- 
viously noted  and  to  document  any  new  deficiencies.  Those 
deficiencies  that  the  shipbuilder  has  responsibility  for  under 
contract  warrantees  are  presented  to  him  for  correction  during 
a subsequent  shipyard  availability.  A description  of  the 
key  T&E  events  for  the  post  delivery  period  of  a surface 
combatant  ship  is  described  in  Appendix  B.  , 


18 


Model  of  the  Ship  T&E  Process 


Figure  4 is  the  model  of  the  flow  of  system/ship  T&E 
phases  which  emerges  from  the  previous  description.  Indi- 
vidual systems  undergo  development  T&E  and  operational  T&E 
independent  of  and,  for  the  most  part,  prior  to  inception  of 
the  ship  acquisition  programs  which  ultimately  use  them 
(first  phase) . When  a ship  acquisition  program  is  initiated, 
and  the  risks  and  uncertainties  associated  with  integrating 
some  of  these  systems  warrant  it,  a land  based  test  site  is 
constructed  for  further  development  and  operational  T&E 
(second  phase) . What  significant  testing  remains  for  the  last 
two  phases  (ship  construction  and  post  delivery)  is  primarily 
production  acceptance  T&E.  It  has  always  been  a conscious 
Navy  policy  to  accomplish  the  necessary  DT&E  and  OT&E  as 
early  in  this  process  as  possible  in  order  to  minimize  the 
necessity  for  nit.  jr  equipment  modifications  and  time  consuming 
retesting  during  the  construction  and  post  delivery  phases 
where  the  cost  and  schedule  repercussions  would  be  significant. 
It  is  this  practice  which  enabled  the  Navy  to  readily  comply 
with  the  try-before-buy  policy.  The  policy's  only  significant 
impact  has  been  in  major  combatant  ship  programs  in  which  the 
decision  to  construct  follow  ships  of  the  class  (DSARC  III) 
is  delayed  until  the  DT&E  and  OT&E  at  the  combat  system  land 
based  test  site  have  been  successfully  completed.  The  only  , 
major  testing  to  be  accomplished  thereafter  is  PAT&E  to 
insure  that  the  shipbuilder  has  fully  met  the  contract 
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specifications  and  that  the  ship  is  in  the  proper  state  of 
material  readiness  for  at-sea  operation. 

Although  most  of  the  testing  on  the  ship  platform  itself 
is  PAT&E,  the  Navy  does  nevertheless  conduct  some  operational 
T&E  as  part  of  the  post  delivery  T&E  phase  of  the  lead  ship 
of  a class.  However,  this  OT&E  is  not  critical  to  the  acqui- 
sition process  in  that  it  is  conducted  well  after  the  commit- 
ment to  production  of  the  follow  ships  has  been  made.  To  date, 
such  OT&E  projects  have  been  assigned  primarily  to  evaluate 
some  aspects  of  system  integration  which  hitherto  had  under- 
gone OT&E  in  a simulated  status  only. 


\ 
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SECTION  IV 


CONSIDERATIONS  OF  CONDUCTING  AN  OPEVAL 

The  desirability  of  conducting  an  OPEVAL  on  a convention- 
al (vice  R&D)  ship  program  has  been  studied  for  two  ships,  viz., 
CGN-38  (formerly  DLGN-38 ) (9)  and  the  FFC-7  (10),  but  in  neither 

case  was  it  found  to  be  cost  effective.  Three  considerations 
follow  to  support  the  conclusion  that,  other  than  in  the 
special  case  of  an  R&D  ship  program,  the  concept  of  a total 
ship  OPEVAL  would  never  be  practical: 

1.  The  inherent  constraints  of  the  test  exercise  would 
in  themselves  introduce  uncertainties  to  and  under- 
mine the  confidence  in  the  validity  of  the  results; 

2.  the  objectives  of  an  OPEVAL  as  applied  to  a ship 
program  are  already  achieved  under  the  existing 
Navy  ship  T&E  process;  and 

3.  even  assuming  that  some  additional  knowledge  would 
be  gained,  it  is  doubtful  that  there  would  be  a 
significant  return  on  investment. 

These  three  considerations  are  discussed  below. 

Constraints 

OPLVALs  as  conceived  in  DOD  Directive  5000.3  and 
OPNAVINST  3960.10  are  highly  structured  evolutions  with 
ccnclusions  drawn  only  after  the  collection  cf  a great  amount 
of  data  from  many  replications  of  combat  situations.  Variables^ 
such  as  environment  and  crew  training  are  assiduously  con- 
trolled so  that  the  result  is  as  clear  a reflection  of  system 
capability  as  possible.  To  undertake  this  type  of  analysis 
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on  a total  system  as  complex  as  a combatant  ship  - f. 
devise,  support  and  conduct  valid  tests  with  sufficii  v.t 
replication  to  overcome  statistical  anomalies  - woulc  be  a 
mammoth  undertaking  involving  the  commitment  of  great  amounts 
of  resources  in  the  form  of  money,  time,  material,  and  fleet 
units.  This  would  require  a scenario  type  exercise  similar 
to  a war  game  or  a training  battle  problem  which,  by  necessity, 
would  be  limited  to  a short  period  of  time.  There  is  con- 
siderable uncertainty  introduced  with  this  approach.  Variables 
such  as  crew  training,  environment,  and  material  readiness 
that  exist  at  the  time  will  influence  the  outcome  and  are 
likely  to  yield  erroneous  conclusions.  Although  the  OPEVAL 
would  seek  to  simulate  a wartime  situation  as  realistically 
as  possible,  various  artificialities  such  as  safety  pre- 
cautions which  preclude  live  firings  would  still  necessitate 
careful  interpretation  of  the  results. 

Furthermore,  it  would  be  difficult,  if  not  impossible, 
to  develop  valid  measures  of  effectiveness  (MOEs)  for  a total 
ship  OPEVAL.  As  described  previously,  combatant  ships  contain 
systems  which  have  previously  been  tested  by  OPTEVFOR  for 
operational  suitability  and  effectiveness  prior  to  their 
integration  into  the  total  ship  system.  The  concern  at  the 
ship  system  level  is  with  system  integration,  information  flow, 
transfer  between  operational  modes,  reaction  time  character- 

\ 

istics,  and  command  and  control.  These  factors  do  not  lend 
themselves  to  evaluation  by  combinations  of  simple  MOEs. 
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A study  conducted  in  1973  on  OT&E  of  ships  expands 
on  this : 

"It  may  well  be  that  measures  of  effectiveness 
no  longer  are  useful  in  evaluating  complex 
weapon  systems  operating  in  an  integrated 
platform  environment  against  an  equally  com- 
plex threat.  The  ABM  debate  is  a case  in 
point.  The  Operational  Research  Society  of 
America  ultimately  convoked  a committee  which 
condemned,  in  print,  the  analyses  and  represent- 
ations of  certain  individuals  of  great  personal 
reputation.  [11:1123-1258],  These  severe 
disagreements  centered,  in  part,  around  the 
assumption  that  simple  MOEs  could,  in  fact, 
represent  the  value  of  such  systems  for  DOD 
and  Congressional  evaluators.  The  seemingly 
endless  controversies  that  occurred  concerning 
the  methodology  for  developing  the  specified 
MOEs  make  this  assumption  questionable." 

(12:111-13) 

Objectives  Achieved  Elsewhere 
The  primary  purpose  of  conducting  an  0PE7AL  is  usually 
to  provide  an  assessment  of  operational  suitability  and 
effectiveness  as  an  input  to  the  decision  to  go  into  pro- 
duction (DSARC  III) . However,  in  some  cases  an  OPEVAL  type 

3 

of  project  is  conducted  after  the  production  decision.  This 
would  be  the  situation  if  an  OPEVAL  were  included  in  a ship 
acquisition  T&E  program,  since  DSARC  III  occurs  more  than  two 
years  prior  to  the  time  the  lead  ship  couxd  undergo  an  OPEVAL. 
In  this  case  the  purposes  for  conducting  an  OPEVAL  are  some- 
what different.  When  applied  to  a ship  program,  the  following 
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Throughout  this  paper,  the  term  "operational  evalu- 
ation" (OPEVAL)  is  used  in  a general  sense  referring  to  any  ' 
OT&E  exercise.  It  does  not  refer  to  the  more  restrictive 
Navy  definition  where  it  is  defined  as  the  OT&E  exercise 
occurring  at  the  end  of  R&D  just  prior  to  the  major  pro- 
duction decision  (Chief  of  N;  'al  Operation’s  Instruction 
3960.10) . 
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six  purposes  for  conducting  an  OPEVAL  are  found  for  the  most 
part  to  be  accommodated  elsewhere.  The  model  of  a ship 
acquisition  T&E  program  developed  in  Section  III  will  be 
used  to  demonstrate  this. 

a.  The  primary  objective  of  conducting  an  OPEVAL  after 
production  initiation  is  to  demonstrate  that  the  system 
development  has  achieved  the  required  operational  capabilities. 
As  discussed  previously,  for  a ship  this  equates  largely  to 
the  sum  of  the  capabilities  of  the  shipboard  systems  and  is 
therefore  known  from  the  individual  systems'  T&E.  Capabilities 
which  cross  major  system  boundaries  can  usually  be  fully 
demonstrated  at  the  LBTL.  One  prominent  reason  why  DDT&E 
withdrew  his  consideration  of  requiring  an  OPEVAL  of 
FFG-7  was  the  success  in  use  of  the  Combat  System  and 
Propulsion  System  LBTSs  to  demonstrate  these  operational 
capabilities  and  to  uncover  deficiencies  that  would  otherwise 
have  gone  undetected  until  at-sea  operation  years  later.  (13) 
These  two  ship  program  LBTSs  were  the  first  ones  constructed 
with  a key  objective  being  the  conduct  of  early  operational 
T&E. 

One  area  that  cannot  be  thoroughly  demonstrated  even 
with  the  hardware  at  the  LBTS  is  how  the  overall  combat 
capability  is  affected  in  selected  degraded  modes  of 
operation,  e.g.,  when  there  are  electrical  or  mechanical  ' 

problems  in  the  interface  between  weapons  systems  and  the 
ship  or  electromagnetic  interference  problems  between  radars. 
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However,  the  possible  permutations  and  combinations  of  such 
degradations  are  such  that  a comprehensive  study  could  only 
be  done  through  the  use  of  model  based  computer  simulation, 
rather  than  through  actual  shipboard  demonstration.  Within 
the  last  several  years  the  Naval  Sea  Systems  Command  has 
developed  and  ul ilized  a Combatant  Capability  Assessment 
Program  in  this  manner  during  the  design  of  the  FFG-7 , the 
CSGN,  and  the  DDG-47  Classes  (14) . 

b.  The  second  purpose  is  to  identify  any  design 
deficiencies  in  order  to  correct  them  or  learn  to  adapt  to 
them.  The  extensive  T&E  efforts  undertaken  prior  to  ship 
construction  have  proven  to  be  adequate  for  this  purpose. 

For  a ship  program  the  DT&E  and  OT&E  of  the  individual 
systems  on  surrogate  platforms  and  of  the  integrated  systems 
in  the  shipboard  configuration  at  land  based  test  si  :s  will 
uncover  any  major  technical  and  operational  deficiencies 
well  in  advance  of  the  time  frame  when  the  ship  OPEVAL  would 
be  conducted.  Also,  there  is  an  exhaustive  series  of  tests 
and  trials  conducted  on  each  ship  during  construction  and  from 
the  time  of  delivery  up  to  turnover  to  the  Fleet  a year  later. 
These  include  Acceptance  Trials  and  Final  Contract  Trials  by 
the  Navy's  Eoard  of  Inspection  and  Survey  as  well  as  numerous 
certifications  and  qualification  trials  for  the  ship's  sonar 
and  radars,  missile  and  gun  systems,  anti-submarine  warfare 
systems,  and  the  propulsion  system.  (See  Appendix  B) . These, 
too,  uncover  any  design  deficiencies  or  installation  errors. 
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The  imposition  of  an  OPEVAL  would  not  obviate  the  need  to 
run  these  exercises  separately,  since  most  of  them  have 
secondary  objectives  beyond  T&E  such  as  system  calibration, 
contractual  acceptance,  and  crew  training.  The  OPEVAL  would 
therefore  be  in  addition  to,  and  in  many  areas  a duplication 
of,  these  trials. 

c.  The  third  purpose  is  to  confirm  the  correction  of 
discrepancies  previously  noted.  This  is  rather  important  in 
a shipboard  system  acquisition  program  where  the  turnover 
from  the  Material  Command  to  the  Fleet  is  abrupt.  In  a ship 
program,  however,  the  turnover  process  is  gradual.  It  is  a 
continuing  evolution  through  the  eight  to  ten  year  course  of 
the  acquisition  program.  During  the  last  year  of  construction, 
the  ship's  crew  resides  at  the  shipyard,  identifying  problems 
that  they  see  with  the  ship  and  trying  to  get  them  corrected 
before  they  have  to  take  over  the  ship  upon  Commissioning. 

Also,  the  identification  and  resolution  of  material  deficiencies 
is  a very ‘high  priority  with  the  ship's  crew  for  the  year  of 
post-delivery/post-commissioning  tests  and  trials.  Any 
marginal  benefits  to  be  gained  in  this  area  from  a ship 

OPEVAL  are  somewhat  debatable. 

d.  The  fourth  purpose  is  to  develop  initial  procedures 

and  tactics  on  how  to  employ  the  system.  In  the  case  of  con- 
ventional ships,  this  is  not  a significant  effort.  The  ‘ 

missions  and  roles  of  the  various  types  of  ships  are  well 
established  in  fleet  doctrine.  In  the  case  of  a brand  new 
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type  of  ship,  such  as  a patrol  combatant  hydrofoil,  the 
development  of  tactics  is  a very  long  evolution  beginning 
with  the  initial  force  level  studies  that  originally 
identified  the  need  for  the  ship  and  extending  through  the 
first  few  years  of  use  and  participation  in  major  fleet 
exercises.  The  contributions  to  this  effort  from  a one  or 
two  month  OPEVAL  would  be  negligible. 

e.  The  fifth  purpose  is  to  identify  key  logistics  con 
siderations  which  should  be  modified  or  established  to  ensure 
that  the  fleet  can  support  the  product.  With  the  exception 
of  training,  logistics  support  is  handled  at  the  equipment/ 
system  level.  Each  respective  system  manager  develops  and 
refines  the  logistics  support  required  for  his  system  as 
well  as  for  its  interface  with  other  systems  on  the  ship. 

The  adequacy  of  this  support  is  later  verified  by  the  ship's 
crew.  In  addition,  these  logistics  support  considerations 
are  continuously  refined  and  validated  throughout  the 
acquisition  program.  For  example,  in  the  combat  systems  area, 
a Logistics  Special  Assistance  Team  comes  aboard  after 
delivery  as  part  of  the  Combat  System  Ship  Qualification 
Trials  of  each  combatant  to  verify  the  adequacy  of  the 
logistics  support. 

In  the  training  area,  there  is  some  team  training  required 
for  the  ship  which  cannot  be  accomplished  within  the  individual 
systems'  areas.  This  is  accomplished  during  the  usual  training 
evolutions  of  the  ship's  post  delivery  period.  Also,  each 
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is  usually  scheduled  to  participate  in  a major  fleet  exercise 
during  the  first  two  years  after  Commissioning.  Any  contri- 
butions to  thesr  training  efforts  from  a ship  OPEVAL  would 
of  course  be  beneficial,  but  in  themselves  would  do  little  to 
justify  the  large  expense  of  the  OPEVAL. 

f.  The  sixth  purpose  is  to  provide  early  feedback  to 
assist  in  the  design  of  future  systems.  For  a ship,  this 
type  of  feedback  must  be  - and  is  - a continuous  effort 
throughout  the  ship's  life.  It  is  provided  by  many  means, 
from  the  formal  Acceptance  Trials  and  Underway  Material 
Inspections  to  the  routine  application  of  the  Planned  Main- 
tenance Sub-system  by  the  sailors  on  the  ship.  The  review 
mechanism  of  the  Fleet  Modernization  Program  is  used 
extensively  to  document,  collect,  and  sort  such  feedback 
information  on  a continuing  basis  throughout  the  ship's  life 
and  to  prioritize  proposed  improvements  for  the  allocation 
of  Operation  and  Maintenance  funds.  Any  feedback  from  a 
one-time  OPEVAL  would  provide  a negligible  addition  to  this 
process. 

Figure  5 summarizes  the  coverage  of  these  OPEVAL 
objectives  during  the  other  portions  of  the  ship  acquisition 
T&E  program. 
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Doubtful  Return  on  Investment 


To  conduct  an  OPEVAL  of  the  CGN-38  has  been  estimated 
by  COMOPTEVFOR  to  cost  between  $3  million  and  $16  million, 
depending  on  what  portion  of  the  participating  fleet  units' 
operating  costs  would  be  charged  directly  to  the  OPEVAL.  (15) 
However,  little  return  on  this  dollar  investment  can  be 
assured  from  the  conduct  of  such  an  OPEVAL.  It  would  provide 
little  feedback  which  would  not  otherwise  be  provided  through 
other  T&E  and  through  fleet  operations  of  the  ship.  The 
constraints  are  severe , which  would  make  many  of  the  results 
inconclusive;  additionally  there  is  no  proven  method  of 
developing  valid  total  ship  test  criteria. 

In  considering  return  on  investment,  the  perishability 
of  any  conclusions  drawn  must  also  be  recognized.  The 
average  life  of  a Navy  ship  is  30  years.  However,  usually 
only  the  hull  and  propulsion  machinery  will  serve  for  that 
lifetime.  Some  updating  of  the  auxiliary  machinery  - either 
additions  or  exchanges  - typically  occurs  once  in  the  ship's 
life.  Because  of  rapid  advances  in  electronics  technology 
and  the  continually  changing  threat,  the  combat  system 
elements  are  much  more  short-lived.  Recent  experience  has 
shown  that  within  the  course  of  ten  years,  the  ship's  entire 
combat  system  will  have  been  replaced.  (12:1-5)  Thus,  ir. 
identifying  a time  frame  over  which  to  amortize  the  cost  of 
an  OPEVAL,  it  must  be  realized  that  any  returns  from  that 
investment  would  be  netted  in  less  than  ten  years  of  operation. 
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which  is  a shorter  time  frame  than  that  required  for  a ship 
acquisition  program. 

For  there  to  be  a significant  return  on  investment,  the 
results  of  the  OPEVAL  should  be  able  to  impact  the  acquisition 
of  the  remaining  ships  in  the  program.  This  cannot  be  done 
without  extending  the  current  acquisition  process.  In  investi- 
gating the  desirability  of  conducting  an  OPEVAL  on  CGN-38 , the 
Chief  of  Naval  Material  reported  that  by  the  time  such  an 
exercise  -could  be  conducted,  analyzed,  and  reported,  two  of 
the  remaining  three  ships  in  the  program  would  already  have 
been  delivered  to  the  Navy  and  the  third,  with  all  of  her 
equipments  already  installed,  would  be  in  the  final  stages  of 
combat  system  testing.  (16)  Even  on  buys  of  large  numbers  of 
ships,  an  OPEVAL  on  the  completed  lead  ship  occurs  too  late. 

For  example,  on  the  30  ship  DD-963  program,  the  earliest  that 
an  OPEVAL  of  the  lead  ship  could  be  completed  is  mid-1976.  For 
the  program  to  proceed  on  schedule,  award  of  the  final  increment 
of  seven  ships  had  to  take  place  in  January  1975,  more  than  a 
year  earlier. 

Any  extensions  of  the  current  acquisition  process,  cn  the 
other  hand,  would  lead  to  substantially  increased  shipyard 
costs  that  would  probably  exceed  the  cost  of  the  OPEVAL  itself. 
The  hiring  and  training  of  the  appropriate  numbers  of  skilled 

* 

personnel  is  a mammoth  undertaking  for  a shipyard.  Any 
significant  time  lag  between  production  of  the  lead  and  follow 
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ships  would  result  in  severe  personnel  disruptions.  The 
shipbuilder  would  be  forced  to  let  most  of  his  experienced 
workers  go  and  rehire  others  when  construction  resumed. 

Much  of  the  "learning  curve"  of  experience  would  have  been 
lost.  Also  a significant  amount  of  the  equipments  for  a 
ship  require  special  fabrication  or  the  use  of  scarce  material. 
Significant  cost  savings  that  are  realized  by  quantity  pro- 
curement of  these  items  would  be  lost  if  a time  lag  were 
imposed  between  lead  and  follow  ships. 

In  summary,  for  there  to  be  a payoff  from  the  conduct 
cf  a total  ship  OPEVAL: 

(a)  it  would  have  to  identify  significant  deficiencies 
that  would  not  otherwise  be  uncovered  by  testing 
in  the  rest  of  the  ship  acquisition  T&E  process; 
and 

(b)  the  value  (translated  into  dollars)  of  the  in- 
creased Fleet  capability  resulting  from  correcting 
these  deficiencies  must  exceed  (1)  the  cost  of 
conducting  the  OPEVAL  ($3  million  to  $18  million 
for  CGrJ-33,  as  discussed  above)  plus  (2)  the 

cost  of  correcting  the  deficiencies  on  the  lead  ship 
of  the  class. 
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SECTION  V 


SUMMARY 

Summary  of  Conclusions 

The  conclusions  of  this  study  project  reinforce  the 

Navy's  position  in  1972  on  the  issue  of  conducting  total 
4 

ship  OPEVALs  . With  much  more  experience  in  implementing 
the  new  policies  on  operational  T&E  and  in  applying  them  tp 
ship  programs  in  particular,  it  would  appear  conclusive  that 
the  conduct  of  a total  ship  OPEVAL  would  be  impractical.  The 
difficulty  of  trying  to  simulate  the  combat  environment  for 
the  ship,  and  the  inherent  constraints  in  any  exercise  which 
would  attempt  to  do  so,  would  introduce  uncertainty  into  and 
undermine  the  validity  of  the  test  results.  Most  of  the 
measures  of  ship  operational  capability  can  be  demonstrated 
elsewhere.  In  particular,  land  based  test  sites  used  in 
ship  programs  have  reached  a state  of  such  development  and 
sophistication  that  they  can  now  satisfactorily  be  used  for 
the  conduct  of  IOT&E.  The  FFG-7  program's  combat  system 
LBTS,  described  in  Appendix  A,  is  a case  in  point.  Little 
if  any  return  could  be  assured  for  the  amount  of  money 
invested  ir.  an  OPEVAL.  In  most  cases,  the  results  of  the 
total  ship  OPEVAL  could  not  impact  the  procurement  of  the 
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Although  DDT&E  has  decided  not  to  require  an  OPEVAL  of 
the  two  programs  studied  to  date  (CGN-38  and  FFG-7) , he  does 
not  consider  these  decisions  as  precedents.  He  will  consider 
requiring  a total  ship  OPEVAL  foi  future  ship  programs  on  a 
case-by-case  basis.  (13) 
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remaining  ships  in  the  program  unless  the  usual  ship  acqui- 
sition process  was  significantly  extended,  resulting  in 
shipbuilding  costs  of  equal  magnitude  with  the  OPEVAL  itself. 

In  addition,  the  results  would  not  be  valid  for  the  entire 
life  of  the  ship,  since  most  of  its  weapons  and  electronics 
systems  will  be  replaced  within  ten  years  after  delivery 
anyway . 

Implicat.'.-  - 

There  are  some  interesting  implications  that  evolve 
from  the  conclusions  of  this  study  project,  as  well  as  from 
the  report  itself.  The  study  project  concludes  that  the 
conduct  of  a total  ship  OPEVAL  is  impractical  in  all  cases 
except  programs  where  a lead  ship  is  constructed  as  an  R&D 
effort  involving  a major  technological  advance  in  the  hull 
or  propulsion  design.  If  this  conclusion  could  finally  be 
accepted  and  the  ominous  paragraph  of  DOD  Directive  5000.3 
imposing  conduct  of  total  ship  OPEVALs  "if  required"  could 
be  deleted,  a major  psychological  obstacle  in  the  minds  of 
many  Navy  planners  in  understanding  the  ship  acquisition  T&E 
process  would  be  removed.  Project  managers  for  new  ship 
acquisition  programs  could  be  confident  that  the  early  plan- 
ning for  their  T&E  program  could  be  done  with  a degree  of 
assurance  that  it  meets  all  of  the  policy  requirements.  The 
threat  of  being  required  by  OSD  to  restructure  the  T&E  program', 
and  perhaps  the  entire  acquisition  program,  to  include  an 
OPEVAL  would  be  removed.  Ship  acquisition  project  managers 
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have  always  been  accustomed  to  thinking  that  the  major  tech- 
nical performance  hurdles  in  their  program  would  be  accomplished 
by  the  time  that  their  land  based  test  site  efforts  were  con- 
cluded. Thereafter,  except  to  uncover  and  correct  production 
discrepancies,  their  efforts  could  be  focused  on  the  Herculean 
tasks  of  keeping  shipbuilding  cost  and  schedules  on  track  and 
providing  for  a smooth  transition  of  the  ship  to  the  Fleet. 

This  perspective  is  supported  by  the  fact  that  ship  acquisition 
is  funded  under  a procurement  appropriation,  vice  under  an  R&D 
appropriation.  But  the  prospects  of  possibly  having  to  conduct 
an  OPEVAL  on  the  lead  ship  a year  after  delivery  has  made  the 
project  managers  feel  uncomfortable  in  thinking  this  way. 

Elimination  of  the  project  managers'  hesitancy  about  total 
ship  OPEVALs  is  not  the  final  solution.  Individual  project 
managers  have  made  many  innovations  to  help  T&E  keep  pace  with 
the  rapidly  advancing  technology  in  the  design  and  integration 
of  shipboard  systems.  Inactivity,  though,  on  the  part  of  the 
Navy  T&E  staff  offices  in  recognizing  the  possibilities  for 
more  widespread  application  of  these  innovations  has  impeded 
advancement  of  the  ship  T&E  process.  Much  still  remains  to  be 
done.  In  the  words  of  a noted  author  and  lecturer,  we  have 
seen  only  the  "tip  of  the  iceberg"  in  the  potential  of  T&E 
as  a management  tool  for  ship  acquisition  programs.  (17:27) 

This  report,  itself,  is  a significant  addition  to  the 
literature  on  Navy  ship  T&E.  The  description  and  model  of  the 
ship  acquisition  T&E  process  presented  in  SECTION  III  were 
culled  from  a survey  of  ship  T&E  program  schedules  and  from 
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interviews  with  T&E  managers  of  both  ship  and  shipboard 
system  acquisition  programs.  They  incorporate  many  funda- 
mental precepts  of  the  ship  T&E  process  that  are  not  yet  well 
documented  in  Navy  directives,  a fact  which  is  not  surprising 
since  T&E  was  not  officially  recognized  as  a key  element  of 
ship  acquisition  program  management  until  early  in  1973  with 
the  publication  of  the  first  Navy  directives  on  the  subject. 

(18)  (19)  Two  guidance  manuals  for  use  by  ship  project  managers 
in  structuring  their  T&E  programs  were  developed  around  that 
time,  (20) (21)  but  have  not  been  updated  since  then.  They 
reflect  the  sophistication  of  thinking  that  has  emerged  from 
the  development  and  conduct  of  ship  T&E  programs  since  early 
1973.  The  lack  of  a unified  description  of  the  ship  acquisition 
T&E  process  can  also  be  seen  if  one  reviews  existing  ship 
programs'  Test  and  Evaluation  Master  Plans.  From  these  it  is 
apparent  that  each  project  manager  has  a different  perception 
of  what  the  ship  T&E  process  is,  how  it  has  evolved,  and  how 
it  can  best  be  tailored  to  his  program.  The  conceptual  frame- 
work of  the  process  presented  in  this  paper,  used  here  for  the 
consideration  of  adding  a total  ship  OPEVAL  to  the  process, 
can  likewise  be  used  for  the  purpose  of  considering  other 
innovations  to  the  T&E  process  and  their  implications  for  the 
overall  ship  acquisition  process. 

I ecommendations 

In  addition  to  deletion  of  the  paragraph  of  DOD  Directive 
5000.3  on  total  ship  OPEVALs  and  an  update  of  the  two  Navy 
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guidance  manuals  on  ship  acquisition  T&E  discussed  above, 
three  possible  topics  emerge  from  this  report  for  further 
study . 

As  previously  described,  total  ship  OPEVALs  are  required 
for  R&D  ship  efforts.  The  first  such  OPEVAL,  results  from 
which  will  be  used  as  an  input  to  the  DSARC  III  decision  to 
buy  follow  ships,  is  scheduled  to  commence  on  PHM-1  within  a 
week  of  submission  of  this  report.  A study  project  parallel 
to  this  one,  investigating  the  utility  of  that  OPEVAL,  would 
be  very  useful.  A model  such  as  that  in  Section  III  for  a 
conventional  program  could  be  developed  to  cover  the  generalized 
R&D  ship  case.  Quantitative  measures  of  utility,  derived  from 
analysis  of  deficiencies  uncovered  during  the  PHM-1  OPEVAL  and 
from  the  exercise  of  generating  the  program  model,  could  hope- 
fully then  be  used  in  planning  future  R&D  ship  T&E  programs. 

The  lack  of  such  measures  in  planning  the  PHM-1  T&E  program 
made  trade-off  analyses  difficult.  For  instance,  some  equip- 
ment development  testing  that  would  ordinarily  have  had  to  be 
completed  prior  to  ship  construction  could  have  been  deferred 
until  and  combined  with  at-sea  testing  on  the  PHM-1,  since 
this  was  a development  T&E  effort  also.  But  it  was  difficult 
to  systematically  pre-determine  what  equipment  testing  could 
be  so  deferred  without  the  existence  of  measures  of  utility. 

It  was  found,  for  example,  that  the  formal  land  based  testing 
program  being  conducted  on  the  foilborne  propulsor  unit  was 
redundant  to  and  as  providing  less  realistic  data  than  that 
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being  routinely  collected  on  the  lead  ship.  The  land  based 
test  program  was  subsequently  terminated  with  an  associated 
dollar  savings.  (22)  If  a valid  model  could  be  developed  and 
utilized  to  measure  the  efficiency  of  such  testing,  more 
efficient  and  1 3s  costly  T&E  programs  for  such  ships  could  be 
assured. 

A study  such  as  this  would  be  a significant  and  large 
undertaking.  Throughout  Section  IV  of  this  report,  emphasis 
was  placed  on  the  lack  of  quantitative  ractors  in  conducting  - 
and  even  in  assessing  the  desirability  of  conducting  - ship 
T&E.  For  example,  in  attempting  to  develop  evaluation  criteria 
for  the  Surface  Effects  Ship,  the  Government  Accounting  Office's 
(GAO's)  new  T&E  group  found  that,  because  the  ship's  key  per- 
formance parameters  were  too  dependent  on  uncontrollable 
variables,  the  criteria  could  not  be  definitized.  The  GAO  has 
since  discontinued  this  effort.  A.  further  example  of  this  need 
for  objective  forecasting  was  DDT&E's  decision  not  to  require 
an  OPEVAL  of  either  CGN-38  or  FFG-7 : in  both  cases  the  con- 
clusion that  an  OPEVAL  would  not  be  cost  effective  was  by 
necessity  based  more  on  subjective  assessment  than  on  a 
decisive  numerical  analysis.  The  DDT&E's  staff  believes  that, 
while  sometime  in  the  future  this  type  of  assessment  can  be 
made  more  objectively,  the  necessary  techniques  are  not  avail- 
able now.  (13) 

A second  possible  area  for  study  is  a cost  benefit 
analysis  of  the  two  FFG-7  land  based  test  sites.  There  was  a 
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consensus  among  those  interviewed  during  this  study  project 
that  the  test  sites  paid  handsome  dividends,  but  as  in  the 
preceding  examples,  this  opinion  is  rather  subjective.  Although 
it  was  not  possible  to  do  so  beforehand,  an  analysis  could  be 
performed  now  using  actual  cost  data  as  well  as  estimated 
values  of  ne  deficiencies  uncovered.  This  would  provide  use- 
ful data  for  future  ship  IOT&E  programs. 

The  third  and  related  recommended  study  would  be  to 
investigate  the  applicability  of  Bayesian  decision  statistics 
in  structuring  a ship  T&E  program.  Bayesian  analyses  have 
been  used  in  developing  expected  dollar  values  of  alternative 
test  approaches  for  small  equipments  and  components.  They 
have  not,  however,  been  used  in  latge  systems  on  which  there 
would  be  a very  limited  production  buy  (such  as  the  four  ship 
CGN-33  Class  program) . The  possibility  of  quantifying  the 
value  of  various  tests  and  alternative  test  programs  for  a 
ship  acquisition  program  has  some  promising  implication  for 
Navy  T&E  arid  the  entire  weapons  acquisition  process. 
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APPENDIX  A 


FFG-7  PROGRAM  COMBAT  SYSTEM  LAND  BASED  TEST  SITE 

A combat  system  LBTS  for  the  FFG-7  acquisition  program 
was  constructed  at  Sperry  Systems  Test  Center  in  Long  Island, 
New  York.  The  FFG-7  combat  system  includes  radars  and  sonar 
for  surveillance,  detection,  acquisition,  and  tracking  of 
targets.  Acquisition  and  tracking  data  are  stored  in  the 
processors.  The  displays  are  the  man-machine  interface  with 
the  processors  and  are  used  to  enter  target  data  and  to  con- 
trol engagements  of  targets.  The  weapon  control  processor 
directs  the  gun  and/or  missile  launcher  in  engaging  the  target. 
The  torpedo  control  system  is  presently  a simplified  stand- 
alone system. 

The  need  for  careful  integration  follows  from  the 
interrelationships  among  the  weapon  control  processor  (which 
generates  the  tracking  data,  and  computes  ballistics  and 
engageability  data) , the  weapon  support  processor  (which 
stores  and  computes  the  tactical  data  necessary  to  control 
the  total  ship  during  engagement) , and  the  displays  (by  which 
the  crew  controls  the  combat  system) . In  order  for  the  combat 
system  to  function,  it  is  necessary  that  the  computer  program 
and  the  equipment  be  properly  interconnected  and  integrated. 
Demonstration  of  this  integration  was  a prime  requirement  to 
support  the  FFG-7  Class  production  authorization  decision 
(DSARC  III) . 

The  LBTS  contained  exact  replicas  of  the  snip  spaces  for 
the  four  principal  compartments  in  which  the  combat  system  is 


located.  These  are:  (1)  the  Combat  Information  Center 
(CIC) , (2)  the  CIC  and  Radar  Equipment  Room,  (3)  the  MK  92 

Equipment  Room  or  Forward  Radar  Compartment,  and  (4)  the 
STIR  Equipment  Room  or  Aft  Radar  Compartment. 

Each  compartment  was  fully  outfitted  with  all  of  the 
cabling,  air  conditioning,  water,  power,  and  ancillary 
equipment  that  it  would  have  in  the  actual  ship.  Live 
equipment  was  installed  where  it  was  required.  Where 
operational  equipment  was  not  required , costs  were  reduced 
by  using  mockups.  Equipment  foundation,  cable  laying,  and 
all  other  systems  were  installed  in  accordance  with  shipyard 
practice. 

In  order  to  provide  a complete  environment  for  the 
combat  system  and  to  exercise  the  combat  system  interfaces, 
equipment  was  installed  to  terminate  or  generate  every  signal 
in  the  combat  system.  It  was  impractical  to  install  a missile 
launcher  and  gun,  so  these  elements  were  simulated  w7ith 
equipment  which  have  the  same  interfaces  and  dynamic  responses 
as  the  real  items.  Equipment  was  provided  to  simulate  those 
ship  motion  inputs  to  the  combat  system  which  come  from  the 
ship's  gyrocompass  and  Speed  Log.  The  missiles  which  would  be 
loaded  on  the  missile  launcher  were  also  simulated.  This 
allowed  the  combat  system  to  be  operated  completely,  as  if  on 
the  ship,  to  search  for,  detect,  and  acquire  air  cargets  and  v 
to  generate  and  simulate  engagements  just  as  in  a battle 
situation.  In  addition,  it  was  recognized  that  it  would  not 

always  be  possible  to  operate  the  radars  and  it  would  not 
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always  be  possible  to  operate  target  aircraft  in  a manner 
which  would  completely  exercise  the  combat  system.  Sperry, 
therefore,  designed  and  constructed  a radar  simulator  which 
provided  synthetic  targets  on  the  operator’s  console  in 
the  same  manner  as  if  they  were  real  targets.  (23:147-154)  • 

In  March  1975,  the  Land  Based  Test  Site  was  formally 
commissioned  and  turned  over  to  the  Navy  for  the  conduct  of 
the  formal  IOT&E  period.  During  this  period,  the  combat 
system  was  operated  exclusively  by  the  Navy  crew.  The  only 
maintenance  performed  by  Sperry  was  that  necessary  to  maintain 
the  equipment  for  the  test  center  or  after  the  Navy  crew  had 
determined  that  the  system  was  inoperable  and  had  identified 
the  subsystem  which  was  at  fault.  The  IOT&E  demonstration 
consisted  of  the  following  tests: 

(1)  Overall  combat  system  reaction  times 

(2)  Acquisition  times  for  various  modes  of  operation 

(3)  Air  surveillance  capability  of  each  of  the  search 
radars 

(4)  Operation  under  simulate!  tactical  conditions, 
including: 

(a)  Control  of  helicopters, 

(b)  Detection  of  airborne  targets  by  use  of  radar, 

(c)  Tracking  and  simulated  engagements  of  air  and 
surface  targets,  and 

(d)  Casualty  mode  operation. 

A report  of  this  IOT&E  period  was  issued  by  COMOPTEVFOR 
(24)  and  the  results  were  presented  to  the  DSARC  in  support 
of  its  consideration  to  approve  the  FFG-7  program  for  production. 
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APPENDIX  B 


SURFACE  COMBATANT  SHIP  POST  DELIVERY  TESTS  AND  TRIALS 


Key  Events  and  Duration 


Purpose 


Combat  System  Ship 
Qualification  Trials 
(8-10  weeks) 


To  demonstrate  through  a series  of 
tests,  exercises,  and  firings  that 
the  combat  system  (gun,  missile, 
radars,  and  communications)  is 
capable  of  proper  performance  and 
that  the  crew  is  capable  of  main- 
taining, adjusting,  and  operating 
it. 


Gun  and  Missile  Structural 
Test  Firings 
(3  days) 


To  confirm  through  live  firings 
that  the  ship's  structure  and  the 
weapons  hardware  has  adequate 
strength,  rigidity,  tightness, 
and  heat  resistance  to  withstand 
shock  and  vibration.  Personnel 
hazards  are  also  checked. 


Shock/Blast  Tests 
(5  days) 


To  determine/verify  the  ability 
of  the  ship  and  its  equipment  to 
withstand  external  shock  blast. 


Anti-Submarine  Warfare 
Weapons  System  Accuracy 
Test 

(5  days) 


To  demonstrate  through  operability 
tests  and  actual  weapon  firings 
the  accuracy  of  the  ASW  system. 


Sonar  Certification 
(5  days) 


To  ensure  that  the  sonar,  as  in- 
stalled, satisfies  established 
equipment  tolerances  and  meets 
specified  performance  standards. 


Ship  Electronic  System 
Evaluation  Facility  Tests 
(2  days) 


TACAN  Certification 
(3  days) 


To  ensure  through  at-sea  tests 
and  calibrations  of  the  commun- 
ication system  antenna  patterns 
that  they  are  installed  and 
operable  at  their  design  cao- 
ability . 

To  demonstrate  TACAN  performance 
for  acceptance  certification  fcv 
FAA. 
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Helo  Certification 
(5  days) 


Fleet  Operational 
Readiness  and  Accuracy 
Checks  (FORACS) 

(5  days) 


To  test  and  certify  that  the 
helicopter  deck  is  suitable  and 
compatible  with  the  assigned 
aircraft  and  meets  prescribed 
safety  requirements. 

To  determine  the  operational 
range  and  bearing  accuracy  of 
ship's  sensors  associated  with 
ASW. 


IFF  Certification 
(3  days) 


To  demonstrate  and  certify 
proper  operation  of  the  Identi- 
fication Friend-or-Foe  system. 


Standardization  Trials 
(5  days) 


Acoustic  Signature 
(10  days) 


To  determine  the  capabilities 
of  the  ship  including  tactical 
maneuvering  (turning  circles, 
acceleration/deceleration  and 
directional  stability  and  con- 
trol) , fuel  economy,  and 
underwater  log  calibration. 

To  determine  a ship's  acoustic 
characteristics  (radiated  noise, 
vibration,  and  structureborne 
noise)  and  establish  its  signature 
profile. 


E-2 


REFERENCES 


1.  DEPSECDEF  memo  to  DOD  Components  of  11  February  1971, 

Subi:  "Conduct  of  Operational  Test  and  Evaluation." 

2.  DOD  Directive  5000.3  of  19  January  1973,  Sub j : "Test  and 

Evaluation. " 

3.  DOD  Directive  5000.1  of  13  July  1971  (reissued  on 

22  December  1975),  Subi:  "Acquisition  of  major 
defense  systems." 

4.  OPNAV  Instruction  3960. 1G  of  22  October  1975,  Sub j : "Test 

and  Evaluation." 

5.  OPNAV  Notice  4720  of  8 September  1975,  Sub j : "Approval 

for  Service  Use  of  the  Fire  Control  System  MK  92 
and  Provisional  Approval  for  Service  Use  of  the 
Gun  Mount  MK  75." 

6.  OPNAV  Instruction  5000.42  of  1 June  1974,  Sub j : "Weapons 

System  Selection  and  Planning." 

7.  NAVMAT  Instruction  3960.7  of  31  May  1974,  Sub j : "Test  and 

Evaluation  of  ship  acquisition.” 

8.  "Test  and  Evaluation  in  Ship  Construction  Contracts," 

Matthew  T.  Reynolds,  Navy  Material  Command 
Procurement  Newsletter,  Quarterly  Review, 

Summer,  1974. 

9.  "Operational  Appraisal  Plan  for  DLGN-38,"  Enclosure  (1) 

to  COMOPTEVFOR  letter  73:IEMD  Ser  C140A  of  15  May 

1974. 

10.  "Operational  Appraisal  Plan  for  FFG-7,"  Enclosure  (1) 

to  COMOPTEVFOR  letter  3*,:JEr  Ser  C280  of  30  October 

1975. 

11.  "Guidelines  for  the  Practice  of  Operations  Research," 

Volume  19,  number  5,  September  1971.  Special 
Issue  of  Operations  Research,  ORSA,  pages  1123-1258. 

12.  "OT&E  for  Ships,"  Final  Report  of  a study  conducted  by 

Booz,  Allen  and  Hamilton  Inc.  under  contract  to  the 
Office  of  Naval  Research;  distributed  by  COMOPTEVFOR 
letter  211 :rog/3930-l  Ser  C392  of  3 December  1973. 

13.  E.E.  Tissot,  Jr.,  RADM,  USN,  Robert  H.  Small,  Capt,  USN 

and  Donal  R.  Petersen;  interview  at  the  office  of 
SECDEF's  Deputy  Director  for  Test  and  Evaluation 
(DDT&E) , 5 May* 1976. 


R-l 


14.  "Combatant  Capability  Assessment:  Status  in  the  Ship  Design 

Process,"  Mrs.  Frances  M.  Prout,  CDR  Robert  C.  Baker, 
USN,  and  Mr.  Henry  J.  DeMattia , Jr., Naval  Engineers 
Journal , June  1974. 

15.  COMOPTEVFOR  let  er  32  I.EMD  3930  Ser  784  of  22  August  1974  , 

Sub j : "Cost  Analysis  of  the  DLGN-38  Operational 
Appraisal . " 

16.  Chief  of  Naval  Material  letter  to  the  Chief  of  Naval 

Operations  of  13  November  1974,  Subj : "Desirability  of 
conducting  an  Operational  Appraisal  of  DLGN-38." 

17.  "Test  and  Evaluation  for  Ship  Acquisition  - the  Tip  of  the 

Iceberg,"  Matthew  T.  Reynolds,  Naval  Engineers 
Journal , February  1975. 

18.  OPNAV  Instruction  3960.9  of  29  January  1973,  Subj:  "Test 

and  Evaluation  for  Ship  Acquisition." 

19.  NAVMAT  Notice  3960  of  1 February  1973,  Subj:  "Test  and 

Evaluation  Policy  for  the  Major  Surface  Combatant  Ships 
Project. " 

20.  "Ship  Life-Cycle  Management  Support  Manual:  Ship  Svstem 

Tost  and  Evaluation,"  NAVSHIPS  0900-060-0240,  . .:ly  1973. 

21.  Test  and  Evaluation  Planning  Guide  for  Naval  Ship  Acquisi- 

tion Projects,"  May  1973,  NAVSEA. 

22.  James  T.  Robinson  III,  Test  and  Evaluation  Director  for 

the  PHM-1  program;  interview  at  Naval  Sea  Systems  Com- 
mand Headquarters,  30  April  1976. 

23.  "Land  Based  Test  Site:  A Tool  for  System  Integration  and 

Test,"  CDR.  F.F.  Underwood,  USN  and  P.B.  Auerbach, 

Naval  Engineers  Journal,  April  1975. 

24.  COMOPTEVFOR  letter  34:tm  3930  (X/S-40)  Ser  S-3  of  8 January 

1976,  Subj:  "Report  on  Project  X/S-40,  IOT&E  of  the 
FFG-7  Combat  System." 

25.  W.  Stark,  LCDR,  USN,  Combat  System  Manager  for  the  CGN-38 

C ass;  interview  at  Naval  Sea  Systems  Command  Head- 
quarters, 15  April  1976. 

26.  G.  McNally,  LCDR,  RAN,  Test  and  Evaluation  Director  for  the 

FFG-7  Class;  interview  at  Naval  Sea  Systems  Command 
Headquarters,  23  April  1976. 

27.  "Ship  Trials  from  Noah  to  Nixon,"  Oliver  E.  Raikko  and  Glenn 

E.  Pribbeno,  Naval  Engineers  Journal,  October  1969. 


R-2 


TW' 


28.  ;’The  Importance  of  Testing  and  Evaluation  in  the  Acquisi- 
tion Process  for  Major  Weapons  Systems,"  Report  by 
the  General  Accounting  Office  of  7 August  1S72 
(OSD  Case  #3389) . 


29.  "Lessons  Learned:  T&E  Guidelines  for  Ship  Systems,"  Volume 
4 of  the  Defense  Science  Board's  Report  of  the  Task 
Force  on  Test  and  Evaluation,  20  November  1973. 


30.  "Phase  II  Advisory  Panel  Report  on  Operational  Test  and 

Evaluation,"  COMOPTEVFOR  letter  211:rog/3930  Ser  898 
of  5 November  1973. 


R-3 


